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interdisciplinary courses reflected a balanced perspective, suggestions that 
an interdisciplinary seminar course on the case study method be offered, 
promoting increased lecture attendance among school team members, and 
focusing efforts during the second year on parental and community involvement 
in the project. During the second year, evaluation also included a review of 
curriculum development efforts and assessment of the project’s impact on 
teacher education students; it was recommended that faculty involvement in 
the development of interprofessional curricula be increased. (Contains 11 
references.) (CH) 



***************************************************************************** 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



4 ED 410 892 



Interprofessional Program Evaluation 1 






Running Head: Interprofessional Program Evaluation Case Study 



An Interprofessional Program Evaluation Case Study: 
Utilizing Multiple Measures to Assess What Matters 



Anne Marie Delaney 

Director of Institutional Research, Babson College 



Correspondence concerning this paper should be addressed to: 

Anne Marie Delaney 
Babson College 
Horn Library 313 
Babson Park, MA 02157-0310 
(617) 239-6481 



BEST COPY AVAILABLE 



Paper presented at the 37th Annual Forum of the Association for Institutional Research, 

Orlando, Florida, May 18-21, 1997 



permission to reproduce ani 
disseminate this material 

HAS BEEN GRANTED BY 



Li 

ERIC 



AIR 



lU I HE EDUCATIONAL RESOUF 
INFORMATION CENTER (ERI( 



2 



U.S. DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

□ This document has been reproduced as 
received from the person or organization 
originating it 

□ Minor changes have been made to improve 
reproduction quality. 



Points of view or opinions stated in this docu- 
ment do not necessarily represent official 
OERI position or policy. 







for Management Research, Policy Analysis, and Planning 



This paper was presented at the Thirty-Seventh 
Annual Forum of the Association for Institutional 
Research held in Orlando, Florida, May 18-21, 1997. 
This paper was reviewed by the AIR Forum Publications 
Committee and was judged to be of high quality and of 
interest to others concerned with the research of higher 
education. It has therefore been selected to be included 
in the ERIC Collection of Forum Papers. 



Jean Endo 
Editor 

AIR Forum Publications 




3 



Interprofessional Program Evaluation 2 



ABSTRACT 

An Interprofessional Program Evaluation Case Study: 

Utilizing Multiple Measures to Assess What Matters 

The purpose of this paper is to present a model program evaluation case study that 
illustrates how program evaluation can contribute effectively to academic and professional degree 
programs; how qualitative and quantitative techniques can be employed to produce valid and 
reliable measures for evaluation studies; and how the role of the institutional researcher can be 
enhanced by assuming the role of program evaluator. The paper, which presents the methodology 
and selected results from an evaluation of a three year interprofessional program, highlights design 
and analysis techniques relevant to a broad range of evaluation studies in higher education. 
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Purpose 

The purpose of this paper is to present a model program evaluation case study that 
illustrates how program evaluation can contribute effectively to academic and professional degree 
programs; how qualitative and quantitative techniques can be employed to produce valid and 
reliable measures for evaluation studies; and how the role of the institutional researcher can be 
enhanced by assuming the role of program evaluator. This paper is based on an evaluation of a 
three year interprofessional program designed to prepare teachers and other professionals to 
provide integrated services for at-risk children. The paper presents the methodological 
approaches, including the rationale, design and analytical techniques, as well as selected results 
from the first two years of the program evaluation. 

Program Description 

The goal of the program on which this evaluation study is based, the FIPSE InServ 
Project, was to transform the teacher education program in order to better prepare future teachers 
to address more effectively the physical, social, and emotional needs of children by working in 
partnership with health/human service and legal professionals. Faculty and students from the 
teacher education program and other professional degree programs in psychology, social work, 
nursing, and law collaborated with school staff, representatives from community agencies, and 
parent representatives to accomplish the following objectives: 

• to articulate new pedagogies, knowledge, skills, and dispositions to assist future 

teachers in identifying and addressing the complex educational, social, 
psychological, and health issues confronting today's children, 

• to build interprofessional instructional teams in teacher education and field 

supervision which model interprofessional partnerships in urban schools, and 

• to establish interprofessional teams of students to engage in field 

work/practicum experiences in schools. 
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Review of the Literature 

The design of this evaluation was based on principles of evaluation theory, on models of 

program evaluation and on writings about evaluation of integrated services. A primary principle 

incorporated in this evaluation is that implementation analysis is essential to understanding what 

actually happens during the administration of a program. Such analysis can facilitate the 

evaluation by identifying the complexities associated with carrying out a program proposal; 

establishing linkages between program goals and actions; manifesting the difficulty of achieving 

program objectives; and providing valuable information for future policy development. 

Considering implementation analysis as a mechanism for establishing connections between 

the setting and achieving of goals suggests the need to question the feasibility of the goals as well 

as the success or effectiveness of the implementation. When objectives are not accomplished, one 

explanation is the claim of faulty implementation, but another plausible explanation may be that 

aspirations were set too high. As Pressman and Wildavesky ( 1973) observe. 

People now appear to think that implementation should be easy; they are, 
therefore, upset when expected events do not occur or turn out badly. We 
would consider our effort a success if more people began with the 
understanding that implementation, under the best of circumstances, is 
exceedingly difficult. They would, therefore, be pleasantly surprised when a 
few good things really happened, (pp. xii,xiii) 

Results from implementation analysis provide essential information for the development of 
sound educational and social policy. In this context, policy might be viewed as a hypothesis 
establishing initial conditions and predicted consequences. Implementation involves the ability to 
achieve the predicted consequences after the initial conditions have been met. Through 
implementation analysis, one can identify any mismatches between means and ends and thus guide 
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the revision of policy for the development of more effective programs in the future. Results from 
the implementation analysis provide the necessary basis for a valid assessment of the project's 
success or failure. One must know what was implemented in order to determine what is effective. 

In addition to the emphasis on implementation analysis, this evaluation also incorporated 
ideas from several different models of program evaluation. The evaluation design included a 
focus on program objectives recommended by Tyler (1949); consideration of the program's 
context, input, process and product proposed by Stufflebeam (1983); attention to the 
stakeholder's perspective advocated by Stake (1990); and the inclusion of both formative and 
summative approaches at different stages of the evaluation (Scriven, 1967). 

Given the nature of the program, this evaluation was also designed to reflect the 
characteristics of integrated services projects. Knapp (1995) recommends that evaluation of 
integrated services projects should reflect the unique and complex nature of these collaborative 
endeavors. Integrated services projects involve collaboration among numerous and diverse 
subjects. Evaluators must study the connections among the subjects which are often difficult to 
measure. Goetz (1993) proposes that the design of the evaluation should reflect that integrated 
services is an ongoing process which requires continuous monitoring. Gomby and Larson (1992) 
concur, stating that integrated services is a comprehensive model and study of it should be long- 
term and concentrated. 

Focusing on the complex nature of interprofessional collaboration. Levin and Greene 
(1994) assert that the nature of the collaborative endeavor implies a context of change, even 
chaos, resulting from changes in relationships, populations served, layers of organizations and 
management as well as changing mandates for professionals. To capture the complex, evolving 
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nature of integrated services projects, evaluations should include diverse methodologies and 
incorporate both process and outcome components. 

Outcome evaluations are recommended to be implemented later in the progression of the 
program, allowing the initial phase of the evaluation of a program to focus solely on process 
evaluation (Burt & Resnick, 1992). Knapp (1995) contends that attributing outcomes to the project 
requires an understanding of the process that produced the outcomes. Consistent with these 
recommendations, the evaluation of this program was designed to reflect and support the project's 
developmental phases moving from primarily a formative, qualitative approach in the first year to a 
more summative, quantitative approach in the third year. 

Many of the evaluation principles and research methodologies presented in this paper 
would be relevant to the evaluation of other higher education academic and professional degree 
programs. Such evaluations represents a unique opportunity for institutional researchers to 
assume new or expanded roles as program evaluators and active participants in academic program 
review and curriculum development. Based on his review of the literature on the role of 
evaluation research in higher education decision making, Shapiro (1986) concluded that evaluation 
is likely to have the greatest impact in the context of academic program evaluation and decision 
making. Feasibility considerations preclude comprehensive institutional evaluations and the 
political milieu of higher education systems suggests that utilization of evaluation results is most 
likely to occur at the academic level of decision making. 
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Evaluation Methodology 

First Year Evaluation Methodology 

While the ultimate goal of this project was to prepare teachers to maximize children's 
potential for academic success by addressing their physical, social and emotional needs, 
accomplishment of this goal required the establishment of new intellectual and institutional 
foundations. Therefore, the first year evaluation focused primarily on the project's activities, 
strategies and documents designed to establish the new intellectual and institutional foundations. 
The nature of this phase of the project and of the evaluation was primarily process oriented. Once 
the foundations were established, the evaluation would shift to a more outcome based assessment. 

The first year evaluation effort was conducted primarily as a formative evaluation to 
monitor the progress and contribute to the ongoing improvement of the project's development. A 
graduate student, serving as an evaluation assistant, assumed primary responsibility for this phase 
of the evaluation. When possible, she attended the event, or watched a videotape, summarized 
and interpreted the issues presented or discussed and noted the number of people participating in 
the event. 

A secondary goal of the first year evaluation was to develop systems for documenting the 
accomplishments of the project. These systems would provide the data required for summative 
evaluations to be conducted at the conclusion of the second and third years of the project's 
implementation. The first year FIPSE InServ Project program evaluation included three 
components: an Implementation Analysis of the Project's Activities and Accomplishments; a 
Process Evaluation on Interprofessional Collaboration; and a Product Evaluation on the 
Development of an Integrated Teacher Education Curriculum. 
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Implementation Analysis . The implementation analysis phase of the evaluation was 
conducted through interviews with various participants in the FIPSE InServ Project; attendance at 
FIPSE interprofessional events; and the collection and review of program materials, 
documentation, and meeting agenda. The questions that guided this phase of the evaluation 
include the following: 

• How did the FIPSE InServ Project team translate the goals articulated in the 

proposal into action during the first year of the implementation? 

• What activities and events did the FIPSE InServ Project sponsor during the first year? 

• What was the impact of the FIPSE sponsored events? How many people 

participated and how many people were influenced by these events? 

Process Evaluation . This phase of the evaluation focused on the process of forming and 
involving various teams in achieving the goals of this project. This process was of paramount 
importance as it related to a fundamental principle of the project; i .e., the ability of individuals 
from different professions and segments of the educational community to collaborate efficiently to 
integrate their knowledge and skills to achieve improved services for children. 

The process phase of the evaluation was completed through observation, interviews, and 
reflection on the various collaborative experiences which occurred during the first year. Results 
of this phase yielded information and insights responsive to the following questions: 

• What efforts were expended to create the interprofessional teams? 

• What challenges were encountered in forming the teams and what 

strategies were employed to cope with these challenges? 

• How effectively did the FIPSE InServ teams function during the first year? 

• What were the common and unique contributions offered by the members of the 

university and the school site teams? 
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• What new intellectual, professional, and practical insights emerged from this 

initial collaborative process? 

• To what extent did university team members draw from and share their 

professional knowledge base during the course of discussion? 

• What progress did the school/agency interprofessional site teams attain in 

articulating the needs that present impediments to children's learning? 



Curriculum Evaluation . This phase of the evaluation involved both an assessment of 
progress achieved in creating a draft, curriculum development and a review of the modified or 
newly created courses focusing on interprofessional collaboration. The draft document was 
intended to articulate the specific knowledge, processes, and dispositions considered crucial in 
preparing future teachers for urban, culturally diverse settings. Criteria employed in evaluation of 
the document included : 

• a comprehensive, well documented statement of each profession's (health, law, 

mental health and social work) knowledge base relevant to early 
identification and prevention; 

• adequate representation of the school site teams' views regarding the multiple 

needs and complex problems confronting students in their efforts to learn; and 

• a synthesis of current research relevant to early identification and prevention. 

The evaluation of the modified and newly developed curricula focused on the degree to which the 
curriculum reflected the ongoing dialogue between the university and school interprofessional 
teams; the priorities and concerns articulated by these teams; and the professional standards of 
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Second Year Evaluation Methodology 

The second year evaluation was designed to be both formative and summative. In 
addition to the methodologies employed during the first year - an implementation analysis of the 
project's professional development and educational programs, a process evaluation on 
interprofessional collaboration, and a curriculum evaluation - the second year evaluation also 
included an assessment of the perceived impact of the project on teacher education students. 

Implementation Analysis . Similar to the first year, the second year implementation 
analysis phase of the evaluation was conducted through interviews with various participants in the 
FIPSE InServ Project; attendance at FIPSE interprofessional events; and the collection and 
review of program materials, documentation, and meeting agenda. The following illustrative 
questions were employed in an evaluation interview with the project director during the 
implementation phase of the second year evaluation. 

- How has the project changed or improved since the first year of implementation? 

- What problems from the first year are you addressing during the second year? 

- What is the quality of communication between university and school team members? 

To what extent do they review and incorporate ideas from each other? and 

Have you observed any improvement in communication during this second year? 

- How have the team's definition of different knowledge bases and the needs of schools 

been used to strengthen the teacher education program? 

- What project goals do you plan to emphasize next year? 

Process Evaluation . Results from this segment of the evaluation documented the 
continued efforts, challenges, and accomplishments of the various university-based and school 
site interprofessional teams. During the second year, this phase of the evaluation was expanded to 
include the project's community outreach efforts. 
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Curriculum Evaluation . The second year curriculum evaluation concentrated primarily on 
the revised and newly created interprofessional courses. The evaluation included reviewing 
course syllabi and interviewing faculty members who were asked to discuss how they 
incorporated the knowledge bases of other professions in their courses and how their courses 
helped students prepare for interprofessional collaboration and integrated services. 

Perceived Impact on Students . Reflecting the transition from implementation to outcome 
based evaluation, the second year evaluation focused on the perceived impact of the project on 
the perceptions, values and knowledge of current education students enrolled in the revised 
courses. Survey questions were administered to these students to elicit their perspective on how 
well their teacher education program prepared them to achieve the following objectives: 

• to design and use evaluative procedures appropriate for students in different age, 

gender, ethnic and cultural groups; 

• to use relevant support systems within and outside the school in order to optimize 

opportunities for teaching and learning; 

• to show respect for the unique developmental and cultural needs of special needs 

children, linguistic and other minorities; 

• to deal effectively with complexity in the classroom resulting from multiple teacher 

roles and the academic, social and moral-ethical dimensions of the classroom; and 

• to cope with diversity in terms of teaching students with different ability levels in the 

same class, from different socioeconomic backgrounds, from diverse racial/ethnic, 

cultural backgrounds, as well as students attending an inner-city school system. 

The same survey questions had previously been administered to alumni/ae who graduated from 
the teacher education program prior to the inception of the FIPSE InServ Project. Using Chi- 
Square and t-test analyses, the responses of current students were compared with those of the 
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Selected Evaluation Findings and Recommendations 

First Year Evaluation Results 



Implementation Analysis 

The first year evaluation, which focused primarily on the project's implementation, 
involved observing, recording and interpreting major project events, including an Interprofessional 
Academic Retreat, an Interprofessional Campus Lecture Series, and Major Public Lectures and 
Urban Seminars. Evaluation findings indicated that the interprofessional academic retreat and 
lecture series provided an opportunity for various constituents - students, professors, and faculty 
members - to explore important themes in interprofessional collaboration and to become more 
aware of the needs of children. The evaluation report contained brief summaries of the project 
events as well as commentaries on how these events addressed issues of critical importance and 
achieved the goals of the project. The following is an illustrative summary of one of the 
interprofesional campus lectures. 

One Child. Many Visions was presented on September 30, 1994 from 1 PM to 3 
PM on the college campus. This was a national teleconference that brought policy 
makers, planners, legislators, administrators, and practitioners from the education 
community together with their counterparts in social work, health and human 
services, juvenile justice, and housing. The goal was to have these professionals 
discuss interdisciplinary collaboration on comprehensive school systems. The 
teleconference included a videotaped segment that focused on interprofessional 
teams at Jackson State University in Mississippi, University of Louisville, 

University of New Mexico, and the University of Washington and how these 
schools have developed interprofessional teams to meet the needs of children and 
their families. During the conference, a toll free number was made available so 
that people watching the conference could call in to ask questions. The conference 
served to help professionals look at ways they can collaborate with other 
professionals to serve children and families. Thirteen members of the college 
community attended. A videotape of the teleconference is available for people 
who could not attend. 
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Process Evaluation 

As noted previously, a major goal of this project was to promote interprofessional 
collaboration. To achieve this goal, project administrators created interprofessional teams at the 
university and school sites. The process evaluation phase of this project included a description of 
the teams, an assessment of their progress; the identification of challenges encountered; and an 
analysis of the issues that emerged from the teams' collaborative efforts. 

During the early stages of a project, one of the primary roles of the evaluator is to make 
meaning out of the "ordinary" efforts to launch the project and to transform this information into 
insights that will be valuable to others undertaking similar endeavors. In this evaluation, the 
author aimed to achieve this goal by including a section of the report entitled. Emerging Issues in 
Interprofessional Collaboration . Findings from this phase of the evaluation yielded insights 
relevant to the following aspects of interprofessional collaboration: intellectual issues related to 
the development of an integrated curriculum; professional concerns associated with the ability of 
individuals from different professions to cooperate effectively; and practical considerations 
concerning the need to arrange convenient meeting sites and form of communication. The 
following sections document the ideas and insights shared on these issues. 

Intellectual Issues . Achievement of the goals of this project requires that the intellectual 
preparation of professionals reflect the philosophy of integrated services. A first step in this 
effort involves engaging faculty in the development of a curriculum based on the integration of 
knowledge from different academic disciplines; this is a primary intellectual issue in educating 
professionals for integrated services. At this time, the intellectual basis for interprofessional 
collaboration has not been formally articulated. The accomplishments of the FEPSE InServ 
Project - the draft document, the lectures, the meetings and the bibliography - are attempts to 
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provide the intellectual background in collaboration; to define the issues and needs of 
collaboration more clearly; and to convince faculty of the benefits of collaboration. 

Professional Concerns . This project was in many ways an experiment in designing and 
implementing professional preparation programs and in delivering educational and human 
services. A critical factor in this experiment was the ability of individuals from different 
professions to cooperate effectively. The first potential obstacle to interprofessional collaboration 
addressed in the FIPSE InServ meetings was lack of trust. This factor may be related partially to 
the organizational structure of academic institutions. Academia is an environment where each 
school competes for resources. Because of this competition, representatives from individual units 
may not be inclined to trust others potentially perceived as competitors. Another factor may be 
the individual and competitive nature of scholarly pursuits. As a consequence of this, faculty in 
one academic discipline or professional school may not be aware of issues or relevant knowledge 
outside their field. The meetings and discussions offered through the FIPSE InServ Project were 
designed to promote a sharing of ideas and to build trusting relationships among people from 
different fields. 

Practical Considerations . The translation of new ideas and the integration of new 
professional relationships into practice require that attention be given to certain practical 
considerations. Two practical issues that emerged during the first year of implementation of the 
FIPSE InServ Project were: the need to establish a convenient meeting site and to develop 
ongoing systems for efficient and timely communication. The FIPSE InServ Project coped with 
these issues in the following ways. First, meeting places were varied to accommodate the 
convenience of different constituents. Second, project participants were encouraged to use the 



ERIC 




Interprofessional Program Evaluation 1 5 



most convenient and timely form of communication. Much of the communication between the 
school and university teams was done individually by telephone and e-mail 
Curriculum Evaluation Results 

A major curriculum development activity during the first year was the development of a 
draft document to articulate new pedagogies, knowledge, skills, dispositions and impediments to 
learning. This document was intended to assist future teachers in identifying and addressing the 
complex educational, social, psychological and health issues confronting today's children. 
Information in the draft document focused on what teachers should know about the collaborating 
professions. 

Project participants also identified four courses for revision: Child Development; Family, 
School and Society; Working with Special Needs Students; and Classroom Assessment. These 
courses were selected because they were required for undergraduate education students and they 
encompass fundamental knowledge, processes, and attitudes critical to future teachers. New 
teaching materials were identified, with a special emphasis on carefully constructed case studies 
that have implications for multiple professions. 

In addition, project staff developed a new course, entitled "The Impact of Psychosocial 
Issues on Learning". This interdisciplinary course related to the goals of the FIPSE InServ Project 
in several ways. First, the course was designed to educate students from different professions on 
the importance of interprofessional collaboration. Second, the course focused on a variety of 
needs that children face, so participants in the class would become aware of children's educational, 
social, psychological, and health needs, and they also would become aware of the importance of 
interprofessional collaboration for addressing these needs. 
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First Year Evaluation Recommendations 

Based on findings from the previously described implementation, process and product 
evaluations, the first year evaluation report presented the following recommendations designed 
primarily to promote improvement and enhance the progress and impact of the project during the 
second and third year of implementation. The recommendations are presented in relation to three 
major project goals: the development of interprofessional curriculum, the dissemination of 
knowledge about interprofessional collaboration, and the establishment of collaboration between 
university and school site teams. The rationale following each recommendation is based primarily 
on evaluation findings indicating need for improvement. These recommendations illustrate how 
negative evaluation findings were transformed into positive, constructive recommendations for 
improvement.. 

Interdisciplinary Courses. 

1. Ensure that interdisciplinary courses reflect a balanced perspective . Interprofessional 
academic preparation is essential to the preparation of future professionals who plan to engage in 
interprofessional collaboration. Such courses need to reflect multiple disciplinary perspectives 
and students should be required to take these courses. To strengthen the collaborative 
experience, such courses might include a course project to be completed by students from 
different professional disciplines. This course related experience would potentially expand the 
interprofessional collaborative training opportunities for students and prepare them for their work 
at the school sites. Also students might develop case studies based on their interprofessional 
collaboration during their practicum. 

The evaluation revealed some variability in the extent to which some faculty members 
were actually revising their curricula and incorporating knowledge bases of other disciplines. This 
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recommendation was designed to encourage the project director to work with faculty to promote 
greater integration of other knowledge bases. 

2. Encourage faculty to integrate the lecture series with their courses . More students may 
attend the lectures if they are required as part of regular course activities. In this way students 
will become more aware of the need for and value of interprofessional collaboration. Results 
from recent alumni/ae research conducted at this institution support efforts to share the goals and 
ideas of this project with all students who are preparing to work with children, particularly in a 
school setting. Based on their own professional practice, our alumni/ae advocate intensive pre- 
professional preparation to enable newly practicing professionals to cope with the critical social- 
psychological challenges they will encounter in serving children at risk. This preparation can only 
be enhanced by attending the interprofessional lectures. 

The lectures potentially provided a rich resource of knowledge and expertise. However, 
the evaluation revealed limited attendance at some of these lectures. This recommendation was 
intended to increase the audience by linking the lectures with course requirements. 

3. Offer an interdisciplinary seminar course on the case study method . Given the 
importance of case studies in interprofessional training, service and evaluation, it may be 
beneficial to offer a series of seminars or workshops on the case study method early in the second 
year of the project. Such a series might address both the intrinsic and instrumental value of the 
case study method for social work, education and program evaluation. Faculty, graduate students 
and other project participants would potentially benefit from such a seminar. 

The case study represents an ideal method for studying interprofessional evaluation. Yet, 
a review of selected students' case studies revealed limited knowledge of the method and a 
presentation of stories rather than analytical case studies yielding valuable insights. 
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Interprofessional Lecture Series. 

4. Increase the publicity about each lecture . If it has not already been done, explore 
various channels of communication in the university - including the faculty biweekly paper and the 
student newspaper. Publicizing the project would result in more professionals attending the 
lectures, and therefore, becoming more aware of the needs of children. 

5. Consider holding some of the lectures, retreats, and workshops at the school sites . 

6. Solicit evaluative data for ongoing program planning . A brief evaluation form might be 
distributed at each lecture. Participants might be asked: What was your goal in attending? To 
what extent was your goal achieved? and How has the presentation influenced your perspective 
on interprofessional collaboration? 

As noted, the evaluation revealed the need for more extensive publicity concerning the 
lectures. This recommendation was intended to promote increased attendance among the school 
team members. Further, project participants were encouraged to engage in the formative 
evaluation by soliciting evaluation data about the lectures for future program planning. 
Collaboration between the University and Schools . 

7. During the second year, focus intensive efforts on parental and community involvement 
in the project . Since so much progress has been made already in achieving the goals of the 
University Collaboration Program, more time and resources might be expended in the second year 
on the Public School Services Program, particularly in attempting to increase parental and 
community involvement in the project. 

8. Increase communication and establish strong relationships with representatives of 
community agencies . The FIPSE InServ Project team should intensify efforts to assure 
professionals in the community service agencies of the project's intent to support, not eliminate. 
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their roles at the school site level. Proposed strategies include communicating more often with 
the agency directors; inviting staff to conferences and meetings; providing compensation for their 
involvement in the project; and strengthening efforts to establish a spirit of mutual trust and 
support. 

The last two recommendations, involving problematic findings of the evaluation, identify 
issues that should be considered in the design of other evaluations. The first issue concerning 
parental involvement illustrates the need to consider the feasibility of accomplishing certain goals. 
With this perspective, one is more likely to develop or modify goals so they will be achievable. 
The second issue, concerning the reaction of community agency members who perceived the 
project as a threat to their role, represents an unexpected finding. Evaluations should be designed 
to uncover unexpected findings. 

Second Year Evaluation Results 

As noted previously, the second year evaluation continued to serve a formative role, 
documenting the project's implementation. . In addition, the evaluation included a review of the 
curriculum development efforts and an assessment of the project's impact on students enrolled in 
the new curricula. 

Implementation Analysis 

The implementation section of the evaluation report included the following summary. 
During the second year of implementation, the FIPSE InServ Project implemented many of the 
recommendations proposed in the first year evaluation report. Notable accomplishments in the 
area of interdisciplinary curricula included a well planned and attended seminar on the case study 
method, the integration of the lecture series and the incorporation of multidisciplinary projects in 
some teacher education courses. The scope and quality of the lecture series were also enhanced 
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with increases in the number of lectures and in the number of distinguished, nationally known 
speakers offering these lectures. In response to the recommendations, FIPSE InServ Project staff 
also increased the publicity for the lecture series and designed and administered a formative 
evaluation instrument to monitor the quality and effectiveness of each lecture. Finally, community 
outreach efforts were expanded significantly during the second year. 

Curriculum Evaluation Results 



The second year curriculum evaluation included a description of the revised and new 
interprofessional curricula and analyses of the extent to which each course incorporated the 
knowledge bases of other professions, promoted interprofessional collaboration and generally 
promoted the goals of the project. The curriculum evaluation focused on various aspects of each 
course, including the course overview, goals, content, projects, and readings. Information was 
obtained from interviews with the faculty and analyses of the syllabi before and after the revisions. 

Interviews with the faculty revealed differing perspectives regarding the impact of the 
project on curriculum development. Therefore, the evaluation report included a section 
documenting the variation in faculty views. An excerpt follows. 

In discussing their courses, faculty members differ regarding the role of the 
FIPSE Project and its impact on the teacher education curricula. As reflected in 
the following comment, one faculty member ascribes a strong influence to the 
FEPSE Project. "FIPSE formed the basis for the course content/structure, lecture 
topics, class papers, projects and activities, speakers and films. All speak either 
indirectly or directly to the project's theme." 

Another professor perceives that the FIPSE InServ Project has had an 
indirect effect through activities that generate discussion and awareness of issues 
related to integrated services. The professor comments, "The atmosphere has 
changed as a result of the interscholastic connections made with social work, law 
and nursing and because of the lecture series and seminars. All of this has 
influenced how people view their teaching and teacher education." 
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Still other faculty members contend that much remains to be accomplished 
to realize the FIPSE InServ Project goals. They recommend that the FIPSE 
Project establish a systematic plan to transform the curricula and support this plan 
with relevant resources such as a list of guest speakers for classes and video 
footage of full-service schools that incorporate integrated service ideas. These 
faculty members encourage FIPSE Project staff to provide ongoing leadership and 
guidance in the development of case studies and other curricula materials relevant 
to interprofessional training and integrated services. 



Student Survey Results 

During the spring semester in the second year of the project, a survey was administered to 
students enrolled in the four FIPSE related courses. FIPSE student responses were compared 
with those of 'non-FIPSE students', i.e., alumni/ae who were enrolled in teacher education 
programs from 1987 through 1993, prior to the inception of the FIPSE InServ Project. The 
common questions, administered to both groups of students, related to how well the teacher 
preparation program prepared students to fulfill multiple teacher roles, to utilize relevant support 
systems, and to respond effectively to the unique and diverse needs of the students they teach. 
Selected results from the comparative segment of the evaluation follow. 

Analysis of individual questionnaire items revealed that, compared with the non-FIPSE 
students, the FIPSE students evaluated their teacher preparation program significantly higher in 

terms of preparing them to assume multiple teacher roles (X^ = 30.94, p ^ .001); to use relevant 
support systems to optimize learning (X^ = 9. 16, p ^ .05); to respect the unique needs of special 
needs and minority children (X^ = 16.55 p ^ .001); and to design and use evaluative procedures 

appropriate for children in different groups (X^ = 17.28 p < .001). 

In addition to the individual item analysis, two scales were created to produce more 
reliable measures of perceived preparation for assuming multiple teacher roles and teaching 
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diverse students. The first scale. Preparation for Multiple Teacher Roles, is based on students' 
ratings of two items: ‘preparation for dealing effectively with complexity from multiple teacher 
roles and preparation for using relevant support systems to optimize learning. The second scale. 
Preparation for Teaching Diverse Students , includes six items referring to the preparation to 
respond to children's unique and diverse characteristics in ability and socioeconomic, racial/ethnic 
and cultural background. The statistical properties of these scales are presented in Table 1. As 
shown, the reliability for the first, two-item scale is moderate, .68, and the reliability for the 
second, six-item scale is high, .88. 

Table 1. Statistical Properties of the FIPSE InServ Project Evaluation Scales 



Scales 


Mean 


S.D. 


Reliability 


No. of Range of 
Items Responses 


Preparation for Multiple Teacher Roles 


3.81 


.06 


.68 


2 


Low - High 
1 -5 


Preparation for Teaching Diverse Students 


3.75 


.30 


.88 


6 


1 - 5 



Table 2 presents group mean ratings and the results of the t-test comparing FIPSE and 

non-FIPSE students' ratings on the scale. Preparation for Multiple Teacher Roles . As shown, the 

FIPSE mean rating is significantly higher than that of the non-FIPSE student rating. 

Table 2. Comparison of Student Group Means on their 
Preparation for Multiple Teacher Roles 





FIPSE Students 


Non-FIPSE Students 


t Value 


X 


4.07 


3.70 


5.34 


S.D. 


(.58) 


(.83) 


(p < .001) 


N 


127 


308 





Table 3 presents FIPSE and non-FIPSE students' mean ratings on the scale. Preparation 
for Teaching Diverse Students . As indicated by the t value, the FIPSE mean rating of 4. 13 is 
significantly higher than the non-FIPSE rating of 3.62. 
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Table 3. Comparison of Student Group Means on their 
Preparation for Teaching Diverse Students 





FIPSE Students 


Non-FIPSE Students 


t Value 


X 


4.13 


3.62 


7.35 


S.D. 


(.60) 


(.79) 


(p < .001) 


N 


128 


311 





Qualitative data were also used to document the perceived effect of the FIPSE InServ 
Project. These data generally supported the quantitative data documenting the perception 
that FIPSE related courses were preparing students to identify and collaborate with other 
professionals to address the complex educational, social, psychological, and health issues that 
confront today's children. Some students, however, qualified their positive evaluation by 
noting that they were in the early stages of their teacher preparation program and would need 
more time, education and experience to be prepared. 

Second Year Evaluation Recommendations 

Recommendations presented in the second year evaluation report identified two areas for 
priority attention: interprofessional curriculum development and parental involvement 
Interprofessional Curriculum Development 

1. The FIPSE InServ Project should increase faculty involvement in the development of 
interprofessional curricula . As noted in the project proposal, course instructors should meet 
regularly with the Project Co-Directors in order to share teaching/leaming experiences regarding 
the FIPSE related courses. Now that the FIPSE InServ Project has created the context for 
interprofessional education and training, it may be beneficial to focus more intensively on specific 
aspects of curriculum development. Initiatives, proposed by some faculty, include creating a 
systematic plan to transform the curricula; supporting this plan with relevant resources, such as a 
list of guest speakers for classes and video footage of full-service schools; and providing ongoing 
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leadership and guidance in the development of case studies and other curricula materials relevant 
to interprofessional training and integrated services. The newly developed interprofessional 
curricula for pre-service teachers should emphasize the application of theory to practice and 
incorporate 'real life' situations in the classroom. 

This evaluation revealed the difficulty of changing curriculum. Feedback from the faculty 
documented the importance of respecting faculty members' unique priorities and concerns for 
their own courses and meeting their expectations for resource support. This evaluation 
reconfirmed that any curriculum development project requires a sensitivity to the issues of 
academic freedom and an appreciation of the diversity of faculty members' views regarding 
curriculum development. 

Parental Involvement 

2. The FIPSE InServ Project should continue to reach out and attempt to increase 
parents' involvement in their children's education . A first step to achieving this goal may be to 
identify existing barriers to parental involvement in the schools. Strategies could then be 
developed to overcome these barriers and facilitate parents' participation in their children's 
education and in school events. Through increased communication and responsiveness to parents' 
concerns, the FIPSE InServ Project might realize the ideal goal of collaborating with parents in 
promoting their children's development. 

Results from the first and second year evaluation documented the extraordinary challenge 
of achieving parental involvement particularly in school settings serving at-risk children. These 
evaluations results suggest that understanding the difficulty of the goal as well as possessing time, 
patience and creativity are essential to achieving progress in engaging parents in school based 
programs. . 
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Discussion 

This paper identifies the following types of issues relevant to the design of other 
evaluations: theoretical and design issues, the construction of valid and reliable measures for 
evaluation and professional challenges one might encounter in conducting the evaluation . 

Theoretical and Design Issues . This evaluation demonstrates the importance of utilizing 
theory in the construction of the evaluation design; incorporating implementation analysis in 
conducting the evaluation; and designing an evaluation to reflect a project's developmental phases 
by progressing from a formative to a summative approach. The implementation analysis 
illustrates how one can transform daily activities and events into a chronicle of accomplishments. 
The process evaluation manifests how a general theoretical or conceptual approach can provide 
an organizing structure and illuminate lessons to be learned from an evaluation. The second year 
evaluation presents surveys as a methodology potentially relevant to the outcome phase of an 
evaluation. Finally, the recommendations illustrate how negative findings can be transformed into 
constructive proposals for improvement. 

Evaluation Measures . With reference to student surveys, case studies and the construction 
of measurement scales, this paper demonstrates how evaluators can develop valid and reliable, 
quantitative and qualitative measures to assess program effectiveness and make significant 
contributions to the development of innovative academic and professional degree programs. The 
development of valid and reliable assessment measures is crucial for the summative or outcome 
phase of an evaluation. 
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Professional Challenges . Finally, the role of a program evaluator presents several potential 
challenges to the institutional researcher. Identifying and addressing these challenges during the 
design stage will enhance the potential for success during the implementation of the evaluation. 

Serving as an internal evaluator, the institutional researcher may encounter both political and 
ethical challenges. For example, findings from the evaluation may identify problems, weaknesses 
or even failure to accomplish major project goals. While the evaluator is ethically responsible to 
honestly report the findings, he or she may do well to anticipate how this might be done while 
protecting the reputation of the project director and staff. Establishing open and continuing 
communication with the project director will enhance the potential for identifying problems early 
and making improvements before the outcome phase of the evaluation. Allowing sufficient time 
for formative evaluation may promote a constructive relationship between the evaluator and the 
project director and, thus, enhance the possibility of continuous improvement throughout the 
project. During the evaluation, problems and potential failures can be handled by translating them 
into constructive recommendations for improvement. 

Other challenges that might be anticipated in the evaluation design include: developing 
strategies to obtain project participants' cooperation with evaluation tasks; preserving the 
confidentiality of project participants who offer feedback to the evaluation; and finally, developing 
innovative and creative assessment approaches to handle unexplained directions of the project or 
unsuccessful attempts at project goals. In conclusion, this paper supports the use of multiple and 
diverse methodologies for evaluation to be selected in accordance with the developmental phase 
of the project's implementation. 
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